1. Field of the Invention
The present invention relates to data communication over a network system, and more particularly, to efficient transfer of data from a server system to a client system over the network.
Portions of the disclosure of this patent document contain material that is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure as it appears in the Patent and Trademark Office file or records, but otherwise reserves all copyright rights whatsoever. Sun, Sun Microsystems, the Sun logo, Solaris, Java, JavaOS, JavaStation, HotJava Views and all Java-based trademarks and logos are trademarks or registered trademarks of Sun Microsystems, Inc. in the United States and other countries. All SPARC trademarks are used under license and are trademarks of SPARC International, Inc. in the United States and other countries. Products bearing SPARC trademarks are based upon an architecture developed by Sun Microsystems, Inc.
2. Background Art
The Internet (or xe2x80x9cWorld Wide Webxe2x80x9d) is used to obtain information from different web sites all over the world. Sometimes users in different parts of the world need to access the same file, requiring the file to be sent to two different places. It is not efficient to send the same data to two or more different places. This problem and others can be understood by the following review of the operation of the Internet and the way in which files are transferred over the internet.
The Internet
The Internet is a worldwide network of interconnected computers. An Internet client accesses a computer on the network via an Internet provider. An Internet provider is an organization that provides a client (e.g., an individual or other organization) with access to the Internet (via analog telephone line or Integrated Services Digital Network line, for example). A client can, for example, read information from, download a file from or send an electronic mail message to another computer/client using the Internet.
To retrieve a file or service on the Internet, a client must search for the file or service, make a connection to the computer on which the file or service is stored, and download the file or service. Each of these steps may involve a separate application and access to multiple, dissimilar computer systems. The World Wide Web (WWW) was developed to provide a simpler, more uniform means for accessing information on the Internet.
The components of the WWW include browser software, network links, servers. and WWW protocols. The browser software, or browser, is a user-friendly interface (i.e., front-end) that simplifies access to the Internet. A browser allows a client to communicate a request without having to learn a complicated command syntax, for example. A browser typically provides a graphical user interface (GUI) for displaying information and receiving input. Examples of browsers currently available include Mosaic, Netscape Navigator and Communicator, Microsoft Internet Explorer, and Cello.
Information servers maintain the information on the WWW and are capable of processing a client request. Hypertext Transport Protocol (HTTP) is the.standard protocol for communication with an information server on the WWW. HTTP has communication methods that allow clients to request data from a server and send information to the server.
To submit a request, the client contacts the HTTP server and transmits the request to the HTTP server. The request contains the communication method requested for the transaction (e.g., GET an object from the server or POST data to an object on the server). The HTTP server responds to the client by sending a status of the request and the requested information. The connection is then terminated between the client and the HTTP server.
A client request therefore, consists of establishing a connection between the client and the HTTP server, performing the request, and terminating the connection. The HTTP server does not retain any information about the request after the connection has been terminated. HTTP is, therefore, a stateless protocol. That is, a client can make several requests of an HTTP server, but each individual request is treated independent of any other request. The server has no recollection of any previous request.
An addressing scheme is employed to identify Internet resources (e.g., HTTP server, file or program). This addressing scheme is called Uniform Resource Locator (URL). A URL contains the protocol to use when accessing the server (e.g., HTTP), the Internet domain name of the site on which the server is running, the port number of the server, and the location of the resource in the file structure of the server.
The WWW uses a concept known as hypertext. Hypertext provides the ability to create links within a document to move directly to other information. To activate the link, it is only necessary to click on the hypertext link (e.g., a word or phrase). The hypertext link can be to information stored on a different site than the one that supplied the current information. A URL is associated with the link to identify the location of the additional information. When the link is activated, the client""s browser uses the link to access the data at the site specified in the URL.
If the client request is for a file, the HTTP server locates the file and sends it to the client. An HTTP server also has the ability to delegate work to gateway programs. The Common Gateway Interface (CGI) specification defines a mechanism by which HTTP servers communicate with gateway programs. A gateway program is referenced using a URL. The HTTP server activates the program specified in the URL and uses CGI mechanisms to pass program data sent by the client to the gateway program. Data is passed from the server to the gateway program via command-line arguments, standard input, or environment variables. The gateway program processes the data and returns its response to the server using CGI (via standard input, for example). The server forwards the data to the client using the HTTP.
A browser displays information to a client/user as pages or documents (referred to as xe2x80x9cweb pagesxe2x80x9d or xe2x80x9cweb sitesxe2x80x9d). A language is used to define the format for a page to be displayed in the WWW. The language is called Hypertext Markup Language (HTML). A WWW page is transmitted to a client as an HTML document. The browser executing at the client parses the document and displays a page based on the information in the HTML document.
HTML is a structural language that is comprised of HTML elements that are nested within each other. An HTML document is a text file in which certain strings of characters, called tags, mark regions of the document and assign special meaning to them. These regions are called HTML elements. Each element has a name, or tag. An element can have attributes that specify properties of the element. Blocks or components include unordered list, text boxes, check boxes, radio buttons, for example. Each block has properties such as name, type, and value. The following provides an example of the structure of an HTML document:
 less than HTML greater than 
 less than HEAD greater than 
. . . element(s) valid in the document head
 less than /HEAD greater than 
 less than BODY greater than 
. . . element(s) valid in the document body
 less than /BODY greater than 
 less than /HTML greater than 
Each HTML element is delimited by the pair of characters xe2x80x9c less than xe2x80x9d and xe2x80x9c greater than xe2x80x9d. The name of the HTML element is contained within the delimiting characters. The combination of the name and delimiting characters is referred to as a marker, or tag. Each element is identified by its marker. In most cases, each element has a start and ending marker. The ending marker is identified by the inclusion of an another character, xe2x80x9c/xe2x80x9d that follows the xe2x80x9c less than xe2x80x9d character.
HTML is a hierarchical language. With the exception of the HTML element, all other elements are contained within another element. The HTML element encompasses the entire document. It identifies the enclosed text as an HTML document. The HEAD element is contained within the HTML element and includes information about the HTML document. The BODY element is contained within the HTML. The BODY element contains all of the text and other information to be displayed. Other HTML elements are described in HTML reference manuals.
Communication and File Transfer on the Web
Files and data are stored at web servers and are made available to clients that log onto the web servers. When a client logs onto a web site, they are actually logged onto a web server. The servers and clients are connected by communication paths, including telephone lines. When a client requests a file or other data, it is transmitted from the server to the client over the communications paths. Clients that are logged onto other servers may be sharing some or all of the communications path. There is a limited amount of data that can be transmitted over the communications path at any one time. The amount of data that can be transmitted over a path is known as the bandwidth of the path. If many clients request data, the bandwidth of the path can be exceeded, so that data transfer becomes slower. Thus, each request for data has an associated cost in bandwidth usage. Where several end-user systems request the same data file from the web-server, the entire data file is transmitted over the transmission network for each end-user system. As the number of users corresponding to each web-server increases, the bandwidth drain and network degradation worsen substantially.
A further disadvantage of conventional data transfer methods is that often the user only needs to view a small portion of the data file, making transmission of the entire data file to the user""s system unnecessary. Further, since the entire data file is transmitted unconditionally, the user must often wait until the entire data file transmission is complete before viewing desired portions of the data. This degrades system performance and negatively impacts the client experience.
There is also a cost associated with the particular data path that is used for communication. A path that includes communication between countries, an xe2x80x9cinternationalxe2x80x9d segment, is more expensive than a path that is domestic. Another problem is that many end-user systems are so-called xe2x80x9cthin-clientsxe2x80x9d with limited memory and storage space, unsuited for receiving and storing large amounts of data. For such clients, receiving large files when only a small portion in needed is inefficient at best and impossible at worse. For example, a report may be a large file (e.g. 200 plus megabytes) and may be placed on a server for access. A user may only need to use a few pages of the report. It would be ineffcient for the user to download the entire report if the user only needs to view a few pages of the report. Not only is there a waste of bandwidth associated with transferring such a large file, but the user would be required to have that much free memory to store the file.
The present invention provides a method of demand based retrieval of a data file comprising pages of data. The method of the present invention can be implemented in a network system comprising a remote host system interconnected to at least one end-user system via a second communication link. The retrieval method includes the steps of: (a) receiving a request from an end-user system for the data file; (b) determining one or more data pages currently referenced by the request; (c) retrieving the one or more referenced data pages from the remote host system; (d) transmitting the retrieved data pages to the end user system via the communication link; and (e) transmitting additional pages as demanded by an end user system.
The remote host system can store a plurality of data files each comprising pages of data, and the retrieval method can further include providing information to said end-user systems for identifying each of said data files on the remote host system, and allowing selection of one or more of said identified data files. At least one end-user system can include display means, and the step of selecting a data file can further include displaying one or more pages of the data file on said display means.
In another embodiment, the method of the present invention can be implemented in a network system comprising a remote host system interconnected to at least one local host system via a first communication link, and one or more end-user systems interconnected to the local host system via a second communication link.
The retrieval method comprises the steps of: (a) receiving a request from an end-user system for the data file; (b) determining whether one or more data pages currently referenced by the request are available in the cache buffer in the local host system; and if so (1) transmitting one or more available data pages from the cache buffer to the end user system, otherwise (2) retrieving the referenced pages from the remote host system to the local host system via the first communication link, (3) storing the referenced pages in the cache buffer in the local host system, and (4) transmitting the referenced pages from the local host system to the end-user system via the second communication link.
The remote host system can store a plurality of data files each comprising pages of data, and the retrieval method can further include providing information to said end-user systems for identifying each of said data files on the remote host system, and allowing selection of one or more of said identified data files. At least one end-user system can include display means, and the step of selecting a data file can further include displaying one or more pages of the data file on said display means.
In another aspect, the present invention provides a demand based paging system comprising: (a) a remote host system storing a data file including pages of data; (b) at least one local host system interconnected to the remote host system via a first communication link; (c) one or more end-user systems interconnected to the local host system via a second communication link; (d) a cache buffer in the local host system for storing a plurality of data pages; and (e) a request processor configured to receive a request from an enduser system for the data file.
The remote host system can store a plurality of data files each comprising pages of data, and said paging system can further comprise a user interface for each end-user system, wherein each user interface is configured to (1) provide information to the corresponding end-user system for identifying each of said data files on the remote host system, (2) allow selection of one or more of said identified data files, and (3) and generate requests for the selected data files. Each user interface can further be configured to display one or more pages of a selected data file on a display means in said corresponding end-user system.
The demand base paging system can further comprise a plurality of local host systems interconnected to said remote host system, and one or more groups of end-user systems corresponding to said local host systems, each of said groups being interconnected to a corresponding local host system. The paging system can also include a central file server configured to provide one or more pages of said data file from the remote host system to the local host system via the first communication link. Each of the first and the second communication links can comprise a plurality of network computers interconnected via transmission lines.
The invention can also be used in an environment that maintains a cache buffer in the local host system for storing a plurality of data pages. In such an embodiment, a cache manager is configured to determine if one or more data pages currently referenced by the request are available in the cache buffer in the local host system, and if said referenced data pages are available in the cache buffer (1) transmitting one or more of said data pages to the end user system, otherwise (2) retrieving the referenced pages from the remote host system to the local host system via the first communication link, (3) storing the referenced pages in the cache buffer in the local host system, and (4) transmitting the referenced pages from the local host system to the end-user system via the second communication link.